1. Field of the Invention
The present invention relates to a face image processing apparatus having means for distinguishing a retinal reflection image from other reflection images using an image processing, for extracting it, and for deciding an eye-state (i.e., opened or closed eye) according to an existing state of the retinal reflection image.
2. Description of the Related Art
One example of such a face image processing apparatus is described in Japanese Patent Application No. 8-5713 filed on Jan. 17, 1996 and assigned to the same assignee. This face image processing apparatus is principally aimed at removing a reflection image at a stage of setting a retinal reflection image candidate region within a binarized image. As conditions for setting a candidate region, a shape characteristic and a mobility vector from a previous screen are used. Also, the apparatus is featured in the provision of a retinal reflection image region extraction means for extracting the left-side and right side retinal reflection images, separately.
FIG. 22 is a configuration diagram showing a stimulated-state or wakefulness detection apparatus disclosed in the above-mentioned Japanese Patent Application No. 8-5713. In this figure, a numeral 1 indicates a person to be detected (i.e., hereinafter simply referred to as a detection subjected person), a numeral 10 indicates a camera, a numeral 11 indicates a lighting unit in the form of an LED device, a numeral 12 indicates a half-mirror, a numeral 13 indicates a LED driving circuit, a numeral 20 indicates a variable-density (light and shade) image memory, a numeral 21 indicates a binarization means, a numeral 22 indicates a binarized image memory, a numeral 23 indicates a characteristics extraction means, a numeral 24 indicates an eye-state decision means, and a numeral 25 indicates an optical glass existence and/or non-existence decision means. With the above arrangement, light irradiated by the LED device 11 illuminates the face of the detection subjected person 1 through the half-mirror 12.
Photographing the face of the detection subjected person 1 by the camera 10, a pupil of the detection subjected person 1 is taken as if the pupil is flashing, as shown in FIG. 2(a), according to reflected light from a retina inside the eyeball. This is because the retina has a character of sending the reflected light back to the same direction as the incident light. Since the reflection image from this retina will be photographed remarkably brighter than other parts of the face, with a binarized process for the photographed light and shade image, an image region with a larger luminance could be extracted as a pupil region. By paying attention to a shape feature of this pupil region (retinal reflection image), the opened and closed states of the eyes can be decided.
FIG. 15 is a flowchart showing a retinal reflection image tracking algorithm of the aforementioned face image processing apparatus. At first, by controlling the camera 10, an output image from the camera 10 is adjusted to an appropriate brightness value (step S1). Next, by mode selection, either one of a search mode for extracting a retinal reflection image for the first time or at other times from a lost-eye state in which an eye is lost or not seen, or of a tracking mode for continuously tracking the retinal reflection image which is being currently in the progress of an extraction is selected (step S2).
In the search mode, setting a search window for the most of an entire face is set (step S3); a region to be a candidate of a retinal reflection image is selected according to a size/shape of the binarized region (step S4); and a retinal reflection image region is extracted from those candidates according to a relationship between the relative positions of the left and right eyes (step S5).
In the tracking mode, the tracking windows is set for tracking the retinal reflection image binarized regions found in the search mode, in the left and right, separately (step S6). Specifically, in order to track the left and right eyes separately, a left-eye tracking window (step S7) as well as a right-eye tracking window are set (step S8). Then, the left and right eye candidates are created in the tracking windows (step S4). That is, the left eye candidate in the left eye tracking window is created (step S9), and the right eye candidate in the right eye tracking window is created (step S10). Next, a retinal reflection image candidate from the left and right eye candidates is selected (step S5), and a decision of a closed eye is made (step S11). That is, if the retinal reflection image of either one of the left or the right could be extracted, then an opened-eye decision is made as an eye state (step S12), and a closed-eye decision is made if no retinal reflection image could be extracted (step S13). When the closed-eye decision continues for a long time (equal to or more than a predetermined time), then a lost-eye state is determined (step S14), and the process returns to the search mode for performing re-extraction once again from the beginning.
FIG. 2(a) represents an original image when the detection subjected person 1 is with the naked eyes, and FIG. 2(b) shows the binarized images thereof.
The above-mentioned face image processing apparatus could, as in FIGS. 2(a) and 2(b), extract the retinal reflection image binarized region 27 according to the shape characteristics such as a degree of squareness (i.e., ratio between side lengths of a rectangular shape) or an area thereof, because a reflection image other than the retinal reflection image 26 rarely appears, and thus could decide the opened and closed states of eyes, correctly.
FIG. 3(a) shows an original image when the detection subjected person 1 is with the eye glasses, and FIG. 3(b) shows the binarized images thereof.
The above-mentioned conventional face image processing apparatus could, as in the Japanese Patent Application 8-5713 Publication, even when the eye glass reflection images 28 exist, with paying attention to a point that a brightness is higher in the eye glass reflection image than in the retinal reflection image, distinguish the binarized regions 29 of the eye glass reflection images and the retinal reflection images, by making the binarized regions 29 to be doughnut shape as shown in FIG. 3(b) according to a binarized threshold value control, thus could extract the retinal reflection images.
Also, the reflection images are limited on the eye glass frame, and will not appear around the eyes.
Since the above-mentioned face image processing apparatus could remove the most of reflection images at a stage of extracting the retinal reflection image, the state with no candidate could be decided as the closed-eye state, finally.
As described above, in the above-mentioned face image processing apparatus, no reflection image other than the eye glass reflection images exist, and when the detection subjected person is with the naked eyes, the binarized images are only the retinal reflection images, and when wearing the eye glasses, the reflection images other than the retinal reflection images could be removed by the brightness difference.
However, with the above-mentioned face image processing apparatus, degradation of a relative resolution due to an expansion of the angle of field of the camera such as shown in FIGS. 4(a) and 4(b), or as in FIGS. 5(a) and 5(b), there were many false detection of the reflection images, since it can not distinguish the retinal reflection images 27 and the reflection images 29 other than them, for a change of a face image such as an appearance of a reflection image other than the eye glass reflection images due to an increased output of the LED.
Particularly, at a time of the closed eyes, the retinal reflection images 27 which have been tracked would disappear on the screen, so that there were instead many false detections of the reflection images 29 whose size/shape are very similar to the ones of retinal reflection images located around the eyes.
Once a false extraction is made, a correct position can not be extracted unless waiting for the next lost state of being seen, the eye-state decisions during that time would be much deviated from the actual opened and closed states of eyes, and thus an accurate eye-state decision can not be made.